Images captured with digital cameras might have noise caused by the characteristics of readout circuits and transmission lines included in Charge-Coupled Devices (CCD) and Complementary Metal Oxide Semiconductors (CMOS).
Furthermore, such images might have out-of-focus blurs and camera shake blurs when captured. Thus, the images captured with a digital camera are degraded by noise due to characteristics peculiar to an imaging apparatus in combination with blurs due to the user's operations in capturing.
In the blurs, an image blur caused by camera shake is referred to as “motion blur” and a blur developed due to out of focus is referred to as “out-of-focus blur”.
Recent increase in demand of high-sensitivity capturing requires the needs of restoring an image degraded by blurs (hereinafter referred to as “degraded image”) to its original image (hereinafter referred to as “true image”) as close as possible. Techniques required in high-sensitivity capturing for obtaining clear, noise-free, and blur-free images are roughly classified into twofold; the technique to enhance sensitivity and the technique to make the exposure time long.
In general, the noise increases as the sensitivity is enhanced. Thus, in the technique to enhance the sensitivity, signals are inevitably overwhelmed by the noise. Consequently, most of the image is affected by the noise.
In contrast, a longer exposure time allows much light from an object to be accumulated, and provides an image having little noise. Hence, the signals are not overwhelmed by the noise. The technique to make the exposure time long, however, causes a problem of camera shakes which develop on the image during the exposure time.
Hence, two techniques have been proposed to overcome the problem due to a longer exposure time. One of the techniques is optical deblurring which involves shifting a lens (See Patent Reference 1, for example). The other technique is to obtain the direction and the size of a blur from an obtained image, and to restore the image by signal processing based on the obtained direction and size of the blur (restoring by signal processing. See Patent Literatures 2 to 4 and Non-Patent Literatures 1 to 7, for example).
The exposure time could be made long in order to ensure sufficient light exposure in a dark environment; however, this would increase the risk of a greater camera shake. In order to overcome a blur using the optical deblurring in such a dark environment, the lens needs to be shifted in a greater a time delay when the lens shifts. Furthermore, there is a physical limit for making the shift range great.
The phenomenon that the camera shake degrades an image from a true image to a degraded image can be modeled below. A function representing luminance of each of pixels in the degraded image is obtained by the convolution of a function representing luminance of each of pixels in the true image with a point spread function representing the blur of the image. In contrast, deconvolution may be performed when the obtained degraded image is to be restored into the true image. The convolution is equivalent to multiplication in a frequency domain. Thus, the restored image is obtained when the degraded image is divided by the PSF in the frequency domain.
In the case where the PSF is assumed to be unknown, the restored image is relatively easily obtained by the deconvolution if the effect of the noise is ignored. In contrast, in the case where the PSF is assumed to be known, the PSF needs to be estimated from the degraded image in order to obtain the restored image.
One of the techniques to estimate the PSF is the sparse coding disclosed in, for example, Non-Patent Literature 1. First, the sparse cording involves obtaining a first restoration result from an initial PSF and a degraded image which are manually provided. Next, the technique involves estimating a PSF assumed to be closer to the true PSF based on the first restoration result and the degraded image, and adjusting the initial PSF using the estimated PSF. Then, the technique involves obtaining a second restoration result from the degraded image using the adjusted PSF. After that, the technique involves repeating the operation of obtaining the Nth restored image from the (N−1)th PSF and the degraded image, and of estimating the Nth PSF from the Nth restored image and the degraded image. Hence, the technique simultaneously achieves the estimation of the PSF and the restoration of the degraded image.
Unfortunately, the technique has a problem in that noise, such as ringing artifacts, develops in the restored image. The ringing artifacts are noise developed in the portion of an image with uniformed luminance (solid portion). The ringing artifacts do not make the solid portion look solid.
FIG. 1A shows an image whose luminance changes in jaggies (a true image near an edge). FIG. 1B depicts a graph which schematically shows a luminance distribution of the true image. FIG. 2A shows a degraded image (blurred image) found near the edge and obtained by a camera capturing the image in FIG. 1A. FIG. 2B depicts a graph which schematically shows a luminance distribution of the degraded image.
Assumed here is the case where a camera shake has developed in a horizontal direction when the image is captured. The degraded image in FIG. 2A has its edges blurred because of a camera shake.
FIG. 3A shows an image into which the degraded image in FIG. 2A is restored by signal processing. FIG. 3B depicts a graph which schematically shows a luminance distribution of the restored image. The restored image in FIG. 3A has a portion with the luminance is periodically varying. Such a luminance variation is the noise referred to “ringing artifacts”. The ringing artifacts develop based on the fact that, in the frequency domain, there is a frequency (hereinafter referred to as “zero point”) whose the amplitude value for the PSF becomes 0 or becomes unboundedly close to 0.
Techniques to solve such a ringing problem are disclosed in Patent References 3 and 4, and Non-Patent References 6 and 7. Rather leaving the shutter open for the entire exposure time, the techniques can reduce a frequency whose amplitude of the PSF becomes zero in the frequency domain, by obscuring light according to a predetermined temporal pattern (coding pattern). Accordingly, the ringing artifacts can be prevented. The techniques disclosed in Patent References 3 and 4, and in Non-Patent References 6 and 7 are referred to as “Coded Exposure Photography”.